Fix: Spring 7 Webflux NoSuchElementError on HttpHeaders#entrySet() by mtomik · Pull Request #4556 · elastic/apm-agent-java

mtomik · 2026-08-21T12:19:03Z

What does this PR do?

Since the original PR is stuck for some time,
I created another one with the same commits + a bit better solution for the core of this issue.

Also when I was running these newer tests I found one more issue:

Spring7ServerFunctionalInstrumentationTest#dispatchError test was failing due to TracedSubscriber not removing the reference of the subscription on cancel() from contextMap. so the solution might be the CancellationAwareSubscription that is just the wrapper calling the discardIf() after cancel.

Is it possible that the issue was there even with older versions of reactor (from older Spring version), but now in that Spring 7 ( reactor 3.8.6 ) it was always failing.

Checklist

cla-checker-service · 2026-08-21T12:19:08Z

💚 CLA has been signed

github-actions · 2026-08-21T12:19:12Z

🤖 GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

mtomik · 2026-08-25T14:38:57Z

thanks for the review 👍

the failed build was due to incompatibility of that sub cancel with java 7. I reworked that using Instrumentation -SubscriptionCancelInstrumentation that will call the discardIf on cancelled subscription

mergify · 2026-08-26T13:51:02Z

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

Queue this pull request

…h webflux

jackshirazi · 2026-08-26T20:01:24Z

run docs-build

jackshirazi · 2026-09-02T18:30:51Z

@mtomik are you able to complete the remaining suggested feedback, I was waiting for this to do a release but if it's not going to be completed soon I'll push this to the next release

This reverts commit 42f3bdd0eb5c54392699d96e6d029cceec232587.

…7-support

mtomik · 2026-09-02T23:35:56Z

I found there one more issue. cancel of the subscription was called before collecting all the headers could be finished and that could cause this exception:

java.lang.NullPointerException: null
at java.util.Objects.requireNonNull(Objects.java:220)
at org.springframework.http.HttpHeaders$CaseInsensitiveEntrySet$CaseInsensitiveEntry.getValue(HttpHeaders.java:2334)
at org.springframework.http.HttpHeaders$CaseInsensitiveEntrySet$CaseInsensitiveEntry.getValue(HttpHeaders.java:2319)
at org.springframework.http.HttpHeaders.lambda$forEach$0(HttpHeaders.java:2030)
at java.lang.Iterable.forEach(Iterable.java:75)
at org.springframework.http.HttpHeaders.forEach(HttpHeaders.java:2030)
at co.elastic.apm.agent.springwebflux.WebfluxHelper.copyHeaders(WebfluxHelper.java:320)
at co.elastic.apm.agent.springwebflux.WebfluxHelper.fillResponse(WebfluxHelper.java:310)
at co.elastic.apm.agent.springwebflux.WebfluxHelper.endTransaction(WebfluxHelper.java:182)
at co.elastic.apm.agent.springwebflux.TransactionAwareSubscriber.cancelTransaction(TransactionAwareSubscriber.java:214)
at co.elastic.apm.agent.springwebflux.TransactionAwareSubscriber.cancel(TransactionAwareSubscriber.java:91)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.cancel(FluxOnAssembly.java:658)

so I fixed it with the correct order + added the same error handling that we got for status code there.

@jackshirazi that should be all from me. if the review of my last changes might block your release, no problem. lets move it to another one then.

SylvainJuge · 2026-09-03T14:46:35Z

I just checked the matching performance in the test standard output, and it's totally acceptable:

   Advice name                                                                                    Type ns            Method ns         
   ExecutorInstrumentation$ExecutorRunnableInstrumentation                                             91,301,814         13,214,102   
   JavaxFilterInstrumentation                                                                          46,153,889            616,531   
   InitServiceNameInstrumentation$JavaxInitServiceNameInstrumentation                                  32,963,871          1,479,452   
   SubscriptionCancelInstrumentation                                                                   25,883,477          8,483,966   
   JavaxServletInstrumentation                                                                         30,973,489            656,024   
   JavaxAsyncInstrumentation$JavaxStartAsyncInstrumentation                                            27,429,023            660,029   
   ExecutorInstrumentation$ExecutorInvokeAnyAllInstrumentation                                         14,844,983         11,137,511   
   RunnableCallableForkJoinTaskInstrumentation                                                         16,824,272          1,212,259   
   ExecutorInstrumentation$ExecutorCallableInstrumentation                                              9,833,095          3,811,010   
   ServletContainerInitializerServiceNameInstrumentation$JavaxInitServiceNameInstrumentation           12,067,723                  0   
   ExecutorInstrumentation$ForkJoinPoolInstrumentation                                                  9,219,421          2,371,271   
   ForkJoinTaskInstrumentation                                                                          8,114,142            558,088   
   JavaxFilterChainInstrumentation                                                                      6,956,189            147,314   
   JavaxRequestStreamRecordingInstrumentation                                                           6,463,642            466,558   
   DispatcherHandlerInstrumentation                                                                     6,393,830            468,880   
   HandlerAdapterInstrumentation                                                                        4,913,156          1,504,565   
   ReactorInstrumentation                                                                               5,121,863          1,232,776   
   JakartaServletVersionInstrumentation$JakartaInit                                                     6,340,990                  0   
   JavaxServletVersionInstrumentation$JavaxInit                                                         4,625,790            579,637   
   JavaxAsyncInstrumentation$JavaxAsyncContextInstrumentation                                           4,688,562            190,258   
   JakartaServletVersionInstrumentation$JakartaService                                                  4,754,681                  0   
   JavaxServletVersionInstrumentation$JavaxService                                                      4,353,296            351,680   
   JakartaServletInstrumentation                                                                        3,074,576                  0   
   WarmupInstrumentation                                                                                2,764,028                  0   
   BodyInsertersInstrumentation                                                                         1,995,783            722,432   
   JakartaFilterChainInstrumentation                                                                    2,679,905                  0   
   SystemSingleEnvVariablesInstrumentation                                                              2,247,976            323,788   
   JakartaAsyncInstrumentation$JakartaAsyncContextInstrumentation                                       2,485,043                  0   
   JakartaAsyncInstrumentation$JakartaStartAsyncInstrumentation                                         2,409,624                  0   
   InitServiceNameInstrumentation$JakartaInitServiceNameInstrumentation                                 2,360,988                  0   
   JakartaFilterInstrumentation                                                                         2,317,672                  0   
   ServletContainerInitializerServiceNameInstrumentation$JakartaInitServiceNameInstrumentation          2,218,153                  0   
   JakartaRequestStreamRecordingInstrumentation                                                         2,167,849                  0   
   SystemAllEnvVariablesInstrumentation                                                                 1,637,550            243,327

This instrumentation uses 3 classes, and none of them is an outlier in the table above.

HandlerAdapterInstrumentation
DispatcherHandlerInstrumentation
BodyInsertersInstrumentation

jackshirazi · 2026-09-03T17:54:17Z

+
+    @Override
+    public ElementMatcher<? super NamedElement> getTypeMatcherPreFilter() {
+        return nameContains("Subscriber").or(nameContains("Subscription"));


I think there are a bunch in reactor that don't use those in the names. Maybe add that as a special case?

Suggested change

return nameContains("Subscriber").or(nameContains("Subscription"));

return nameStartsWith("reactor.")

.or(nameContains("Subscriber"))

.or(nameContains("Subscription"));

Asking an LLM to find known implementations of Subscription yields the following list, so reactor. is a good prefix, we might also add io.reactivex. and io.smallrye.mutiny. but I'm definitely not familiar with any of those reactor frameworks. Here it would not hurt to have a few false positives rather than lots of false positives creating lots of useless overhead, so this should be fine as a first step, and we can refine further if needed.

reactor.core.publisher.FluxInterval$IntervalRunnable reactor.core.publisher.FluxSwitchMap$SwitchMapInner reactor.core.publisher.SinkManyUnicastNoBackpressure io.reactivex.internal.operators.flowable.FlowableAmb$AmbCoordinator io.reactivex.internal.operators.flowable.FlowablePublishMulticast$OutputCanceller io.smallrye.mutiny.operators.multi.MultiFlatMapOp$FlatMapInner io.smallrye.mutiny.operators.multi.MultiCombineLatestOp$CombineLatestCoordinator io.smallrye.mutiny.operators.multi.MultiGroupByOp$State io.smallrye.mutiny.operators.multi.MultiZipOp$ZipCoordinator io.smallrye.mutiny.operators.multi.processors.UnicastProcessor io.smallrye.mutiny.operators.multi.MultiOperatorProcessor io.smallrye.mutiny.operators.multi.builders.IntervalMulti$IntervalRunnable org.springframework.http.server.reactive.ChannelSendOperator$WriteBarrier org.springframework.http.server.reactive.ChannelSendOperator$WriteCompletionBarrier

one drawback of this additional name matching is increased instruction time spent. So I run the numbers again:

current filter

2026-09-04 00:03:18,209 [ForkJoinPool-2-worker-1] DEBUG co.elastic.apm.agent.bci.InstrumentationStatsLifecycleListener - Total time spent matching: 257,422,186ns | Advice name | Type ns | Method ns | | ExecutorInstrumentation$ExecutorRunnableInstrumentation | 46,278,675 | 5,203,226 | | SubscriptionCancelInstrumentation | 17,952,170 | 11,488,096 | | JakartaFilterInstrumentation | 24,123,333 | 0 |

OR starts with "reactor."

2026-09-03 23:59:18,358 [ForkJoinPool-2-worker-1] DEBUG co.elastic.apm.agent.bci.InstrumentationStatsLifecycleListener - Total time spent matching: 282,551,157ns | Advice name | Type ns | Method ns | | ExecutorInstrumentation$ExecutorRunnableInstrumentation | 48,814,752 | 6,779,428 | | SubscriptionCancelInstrumentation | 42,561,353 | 8,489,875 | | JakartaFilterInstrumentation | 23,363,543 | 0 | | JakartaAsyncInstrumentation$JakartaStartAsyncInstrumentation | 20,613,139 | 0 |

OR all 3 starts with you mentioned

2026-09-04 00:02:17,662 [ForkJoinPool-2-worker-1] DEBUG co.elastic.apm.agent.bci.InstrumentationStatsLifecycleListener - Total time spent matching: 289,889,601ns | Advice name | Type ns | Method ns | | SubscriptionCancelInstrumentation | 47,053,567 | 14,306,418 | | ExecutorInstrumentation$ExecutorRunnableInstrumentation | 48,076,345 | 5,605,395 | | JakartaFilterInstrumentation | 24,638,337 | 0 | | JakartaAsyncInstrumentation$JakartaStartAsyncInstrumentation | 22,838,079 | 0 |

if we want to optimize the filter even further - with something like just nameEndsWith("Subscriber") we cover most of them with just this overhead.

2026-09-04 00:12:48,681 [ForkJoinPool-2-worker-1] DEBUG co.elastic.apm.agent.bci.InstrumentationStatsLifecycleListener - Total time spent matching: 259,733,930ns | Advice name | Type ns | Method ns | | ExecutorInstrumentation$ExecutorRunnableInstrumentation | 62,550,712 | 5,650,035 | | JakartaFilterInstrumentation | 24,774,004 | 0 | | SubscriptionCancelInstrumentation | 14,498,788 | 9,413,708 | | JakartaAsyncInstrumentation$JakartaStartAsyncInstrumentation | 20,837,849 | 0 |

the nameContains("Subscription") was there just to match it even when someone did his own implementation - bigger chance that we catch it

nameEndsWith("Subscriber") OR all 3 your package prefixes

2026-09-04 00:09:40,953 [ForkJoinPool-2-worker-1] DEBUG co.elastic.apm.agent.bci.InstrumentationStatsLifecycleListener - Total time spent matching: 281,649,260ns | Advice name | Type ns | Method ns | | SubscriptionCancelInstrumentation | 38,772,826 | 14,850,740 | | ExecutorInstrumentation$ExecutorRunnableInstrumentation | 48,154,398 | 5,293,119 | | JakartaAsyncInstrumentation$JakartaStartAsyncInstrumentation | 24,675,265 | 0 | | JakartaFilterInstrumentation | 23,984,791 | 0 |

the thing is that if we miss some of them, we just rely on GC to clean it up later. so it just about finding that balance I guess 😄 but at the end, every one of these will be better than it was before, where we missed all of them

mtomik requested a review from a team as a code owner August 21, 2026 12:19

github-actions Bot added agent-java community Issues and PRs created by the community triage labels Aug 21, 2026

mtomik changed the title ~~Fix/webflux spring 7 support~~ Fix: Spring 7 Webflux NoSuchElementError on HttpHeaders#entrySet() Aug 21, 2026

mtomik force-pushed the fix/webflux-spring-7-support branch from 09b0d62 to 4fb8061 Compare August 21, 2026 12:33